Papers with human parity
Statistical Power and Translationese in Machine Translation Evaluation (2020.emnlp-main)
Copied to clipboard
| Challenge: | a recent paper argues that translationese has been used to describe features of translated text . a translationed text can be more explicit than the original source, authors say . authors recommend reverse-created test data be omitted from future evaluations . |
| Approach: | They propose to omit translationese from future machine translation evaluations . they also re-evaluate a past evaluation claiming human-parity of MT . |
| Outcome: | The proposed analysis shows that translationese does not affect machine translation evaluations. |
Appraise Evaluation Framework for Machine Translation (C18-2)
Copied to clipboard
| Challenge: | Appraise is an open-source framework for crowd-based annotation tasks . it is used for shared tasks at the conference on machine translation and at IWSLT 2017 . |
| Approach: | They present an open-source framework for crowd-based annotation tasks . they describe the entire lifecycle of an Appraise evaluation campaign . |
| Outcome: | The proposed framework is used to run evaluation campaigns at the WMT Conference on Machine Translation and at IWSLT 2017 . it has been adopted by the translator team at Microsoft Translator for internal quality monitoring . |
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for learning audio-text connections rely on parallel audio- text data . a new approach allows for the representation of environmental soundscapes without using parallel data - a challenge for many applications . |
| Approach: | They propose a model that induces Audio-Text alignment without using parallel audio-text data. |
| Outcome: | The proposed model outperforms the current state-of-the-art for audio classification tasks with no audio-text data by 2.2% on the ESC50 and US8K tasks. |
Measuring Robustness for NLP (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to evaluate NLP models are limited to news domains and cannot be generalized to other domains. |
| Approach: | They propose a measure of NLP quality based on robustness . they measure consistency of cross-domain accuracy and introduce coefficient of variation and gamma-Robustness based upon human evaluation . |
| Outcome: | The proposed approach shows higher agreement with human evaluation than accuracy scores on ranking machine translation systems. |
Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels (2023.acl-long)
Copied to clipboard
| Challenge: | Several recent papers claim to have achieved human parity at sentence-level machine translation. |
| Approach: | They propose to use a dataset with rich discourse annotations to evaluate MT performance . they find that MT outputs differ fundamentally from human translations in terms of latent discourse structures. |
| Outcome: | The proposed dataset builds upon the large-scale parallel corpus BWB . it covers 15,095 entity mentions in both languages and compares them to human translations . |
What’s “up” with vision-language models? Investigating their struggle with spatial reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work has re-surfaced a concern that has long plagued vision-language models: poor performance on simple tasks like attribute attachment, counting, etc. |
| Approach: | They evaluate 18 vision-language models and find they perform poorly on VQAv2 . they find that popular vision-linguistic pretraining corpora lack reliable data for learning spatial relationships . |
| Outcome: | The new models are compared with existing datasets on what'sup and visual-language models . they achieve 56% accuracy on the new benchmarks compared to 99% for humans . |
On “Human Parity” and “Super Human Performance” in Machine Translation Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | In this paper, we reassess claims of human parity and super human performance in machine translation. |
| Approach: | They reassess claims of human parity and super human performance in machine translation . they argue that human translation involves much more than what is embedded in automatic systems . |
| Outcome: | The proposed results show that human translation involves much more than what is embedded in automatic systems. |